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— - (57) Abstract: A Process Automation System (figure 1) and accompanying method provides for the aototoated or semi-automated 
creation of workflow management procedures having an arbitrary number of tasks to be performed manually or via computer au- 
tomation in specified sequences. Workflows are automatically or semi-automatically generated by prompting the user with valid 

^ choices. After creating the workflow, the system then acts as the executive for execution; initiating casks, tracking status, and man- 
aging intermediate and final products. The capability is provided to define new products and tasks for the system via a graphical 

^ user interface. 
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WORKH.OW SYSTEM AND BUILDER OFFERS TMAGE SCRIPT TOOLS ON IKPUT DATA TYPE 

Field of the Invention 
This invention relates generally to workflow management wherein computer 
software is used to implement the automated control of multiple-step, non-real-time 
processes. More particularly, this invention creates and executes custom processing 
5 flows by relying on a knowledge-based system rather than user understanding. 

Background of the Invention 
In many sophisticated product development processes, execution and oversight of 
the process is a significant portion of the production effort. Operators must be trained to 
understand production procedures and the use of individual tools. Supervisors must track 
10 and schedule production tasks. File and disk space management has been estimated to 
consume as high as 80 percent of the production labor time. 

Workflow management is used to automate production systems. Workflow 
management has two major sub-fields, control of standard practices and creation of 
custom processing flows. Workflow management of standard practices is commonly 
15 applied to business processes, and generally consists of software in which one or more 
workflow procedures are embodied in the software. Workflow management of custom 
processing flows generally consists of systems to build procedures firom a collection of 
tools. Such systems are commonly found in commercial image processing packages 
(Khoros, Erdas Imagine, AVS), and rely on the user to understand the processing and 
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tools being used. Such image processing systems do not generally support the status and 
tracking needed for production. 

Alternatively, workflow management can be performed manually, using tools for 
tracking status. When done manually, however, operators must track and record the 
5 location of all intermediate files, and production managers must control the movement of 
data between the different operating stages. 

All of the existing methods for workflow management of custom processing 
flows require a great deal of knowledge and/or training on the part of the users. The user 
must be aware of the availability, use, and caveats of each software tool to be 
1 0 incorporated in the workflow procedure. 

Depending upon the particular method b^ing used, the user may also be 
responsible for tracking filenames of intermediate products, recording completion status 
information, and moving data between configured areas. The effort required to do this 
has been measured to be up to 40 percent of the total labor cost of the production. 



15 Summary of the Invention 

This invention was developed in response to the needs set forth above, but with 
the recognition that production processes are constantly evolving due to the introduction 
of new tools and new products. The resulting Process Automation System according to 
the invention automatically creates and executes workflow management procedures for a 

20 variety of applications. Not only does the system handle lower-level tasks such as 
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bookkeeping and file shuffling, it can also be used to dynamically develop new 
production procedures and guide the operators through them once created. 

Broadly, the effectiveness of the process is twofold, in that it automatically or 
semi-automatically creates the workflow " management procedure, and executes the 
5 procedure in an environment that provides the status, tracking and control needed for a 
production environment. 

The system includes three main components: 

• Data and Tool Definition; 

• Work Order Creation; and 
10 • Work Order Execution. 

In the Data and Tool Definition component, software tools and their inputs and 
outputs are defined to the system via a graphical user interface. This provides the 
information that the system needs to reason over how to create a workflow management 
procedures and dynamically generated user interfaces. 

15 In the Work Order Creation component, the creation of the workflow 

management procedure is initiated by the user identifying the desired output product and 
the desired input products. The user may also provide the name of a previously stored, 
abstracted template to provide guidance and eliminate the need for some interaction in 
the semi-automatic operational mode. The Work Order Creation component recursively 

20 reasons backwards fi-om the output product to the input products, determining what 
processing steps and intermediate products are needed to perform the processing. When 
multiple vahd options exist, the user or the abstracted template is consulted for guidance. 
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All prompting for inputs is done via dynamically generated graphical user interfaces, 
based upon the information entered to the data and tool definition component. Once a 
valid processing path exists for a single input, that path can be automatically replicated 
for similar inputs. 

5 Once created, Work Order Execution is invoked to execute the series of tasks, and 

to monitor and control status. The Work Order Execution component is largely based 
upon the WPEP Scheduler, built by Russell Sieron in 1996. Additional capabilities were 
created as part of this development that interface with the Process Automation System 
database and partition workflow management procedures into blocks of processing steps. 

10 Brief Description of the Drawings 

FIGURE 1 depicts the logical structure of the system when used with the 
graphical user interface to create and execute a Workflow management procedure; 

FIGURE 2 depicts the logical structure of the system when run without the 
graphical user interface for workflow management procedure creation; 
15 FIGURE 3 shows a Product and Tools Registration step that provides a graphical 

user interface for various operations; 

FIGURE 4 is set of diagrams which illustrate the atomic definition of tools; 

FIGURE 5 shows how recursive reasoning may be used to build a workflow 
management procedure; 

20 FIGURE 6 shows a workflow management procedure for a single input image; 

and 
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FIGURE 7 illustrates a workflow management procedure following replication of 
a branch. 

Detailed Description of the Invention 
The Process Automation System according to the invention provides a software 
5 utility for the creation and execution of workflow management procedures. It is useful in 
a multitude of applications, and is currently being applied in the field of image processing 
and geospatial information production (under the tradename GeoWorx""). A current 
implementation consists of C and JAVA source code designed for use in the UNIX 
operating system environment It is currently implemented for Solaris 2,6 on a SUN 
10 workstation. The system integrates the following subsystems: 

1. Manual Work Order Creation - the user interface for creation and 
execution of workflow management procedures and abstracted templates (called 
profiles); 

2. Event Driven Work Order Creation - a conunand-line interface for 
1 5 creation and execution of workflow management procedures; 

3. Work Order Interpreter - the logic engine that reasons over needed inputs 
and outputs and available tools to create them, and creates the workflow management 
procedure; 

4. Run Control - the runtime control engine that analyzes the workflow 
20 management procedure for parallelism and creates the commands to execute the tasks; 
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5. Scheduler - the process management control executive that oversees 
process execution on each host, provides real-time and logged status, and provides for 
user intervention to reschedule or terminate jobs; 

6. Database - a centralized database that holds the knowledge base for 
5 reasoning, catalogs of available input data, and processing logs; 

7. Task Support - utilities for conveying status information back to Run 
Control; and 

8. Product and Tool Registration - an application that provides a graphical 
user interface for registering new product types and tools in the system. 

10 Among other differences to prior-art approaches, the Work Order Interpreter 

subsystem can be used to create a workflow management procedure by reasoning over 
the capabilities and dependencies of tools that perform specific tasks. Given the 
specification of the desired product and input(s), the Work Order Interpreter searches a 
database of known tools to determine the needed tools and their sequences to create the 

15 product from the inputs. Supported workflow procedure topologies include linear 
sequences, trees, and lattices. The Work Order Interpreter can be run in fully automatic 
mode, in which defeult selections are always made, or in semi-automatic mode, in which 
the user is allowed to choose among the valid choices available. Workflow management 
procedures can be stored in an abstracted form and recalled for later use with new inputs, 

20 even if the number of inputs has changed. 

Figure 1 depicts the logical structure of the system when used with the graphical 
user interface to create and execute a workflow management procedure. The user 
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interacts with an application that utilizes the Manual Work Order Creation, Work Order 
Interpreter, and Database subsystems. The user selects the desired type of output product 
and the input data from which to make it using the graplucal user interface of this 
application. The user may optionally select to use a pre-stored abstracted template 
5 (profile) to provide further direction and defaults to the Work Order Interpreter. 

The Work Order Interpreter then reasons through the creation of the workflow 
management procedure, stopping only to obtain inputs for information specific to the job 
(i.e. for what part of the Earth should the output product be made) or to resolve 
ambiguities if a profile is not provided. Once the potential workflow management 

1 0 procedure has been completed, the user is allowed to review and modify the procedure as 
desired. The workflow management procedure is then stored in the database and the run 
control application is spawned. 

The run control application obtains the workflow management procedure from the 
database and analyses it for dependencies and parallelism. As each group of tasks that 

15 can be executed in parallel is reached, the control information in the workflow 
management procedure is converted into the proper commands to execute the task. The 
commands are then submitted to the scheduler for execution. The run control application 
is invisible to the user. 

The scheduler manages the queue of submitted tasks on each host, ensuring that 

20 no more than the allowed number of concurrent tasks are being executed. The scheduler 
provides status information about the queue via a graphical user interface. This interface 
allows the user to view the stderr and stdout output of tasks, postpone tasks, and 
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terminate tasks. Upon completion of each task, this stderr and stdout output is e-mailed 
to the user 

The task itself is run via a "task script," a shell script that acts as a wrapper to 
allow any application that can be run without a graphical user interface to be integrated 
5 into the system. The task script utilizes a UNIX command, StatusNotify, to transmit 
status information directly back to the run control application. Any application that 
cannot be run without its graphical user interface can be run as a "Manual Application.** 
When a manual application is found within a workflow management procedure, e-mail is 
sent to the individual who was specified to perform the task when the workflow 

10 management procedure was created. The e-mail contains all of the instructions relevant 
to performing the task, such as the location of input files, the desired location of output 
files, and any special processing notes. The individual runs the graphical user interface to 
StatusNotify before processing begins. This confirms receipt of the message. The 
individual then performs the processing and runs StatusNotify to report the completion 

IS status. 

Figure 2 depicts the logical structure of the system when run without the graphical 
user interface for workflow management procedure creation. The system is accessed via 
a conunand-line firont-end that can be spawned firom another process. All needed 
information is provided on the command line. The remainder of the processing proceeds 
20 as described above. 

The Product and Tools Registration is used by software domain experts to register 
new product types and tools within the system. The logical flow of this capability is 
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depicted in Figure 3, which shows a Product and Tools Registration involving an 
application, AddProcedures, that provides a graphical user interface for all of these 
operations. A product is registered as a data type and the tools that are capable of 
creating that data type. A data type can also be registered to a data class, so that it can be 
5 recognized by tools that work on a variety of input data types. Examples of products 
could include orthorectified images, mosaics, change detection images, and cotmtless 
other imagery and non-imagery products. Tools are registered by providing the name of 
the task script to execute the tool, a categorization of how to invoke the tool (command 
line, manual, prompt-and-respond, etc.), and a detailed description of the inputs, outputs 

10 and control parameters for the application. The language for making this description 
includes support for conditionally needed information, lists of informaition, multiplicity, 
and any permutation thereof. The user sees the language expressed as a tree, with 
graphical control of the tree layout and graphical user interface forms to provide the 
details for each tree node. In the process of defining the interface for the tool, the data 

15 type or class of all inputs and outputs is provided. This provides the information over 
which the Work Order Interpreter reasons. Task scripts are created using a standard text 
editor. 

The Work Order Interpreter views each tool (procedure) as an atomic unit that 
outputs a specific type or class of product when provided with specific types or classes of 
20 inputs. The multiplicity of inputs and outputs is a characteristic of the tool. The Work 
Order Interpreter begins by considering the desired output product. It determines what 
tools are available to make the product. If multiple choices exist, it selects one based 
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upon either defaults, the abstracted template (profile) provided it, or user intervention. 
The method of selecting is based upon the settings of three flags to control its processing. 
The selected choice is then examined to determine its needed inputs. This processing 
continues recursively until either a specified input item is found to fulfill the needed 
5 inputs, or it is determined that the product cannot be made from those inputs. 

Additional ancillary inputs can be specified as needed. This processing is depicted 
in Figures 4 and 5. Once a processing flow has been established for a single input, then 
the remaining inputs are inspected to determine if they are of the same type. If one or 
more inputs of the same type exist, then the user is given the option of replicating the 

10 processing flow branch for the corresponding inputs. When a processing flow branch is 
replicated, each processing step is evaluated to determine if it can accept additional inputs 
and outputs or if a new instance of that processing step must be created. This processing 
is depicted in Figures 6 and 7. The resulting workflow management procedure can be a 
single sequence, a tree, a lattice, or any permutation of the preceding. 

1 5 Some advantages of the Process Automation System described herein include: 

# Automated reasoning guides the user through creating the workflow management 
procedure, greatly reducing the need for training and expert operators; 

• Abstracted templates can store workflow management procedures for reuse, 
recalling preferences and processing structure; 

20 • Abstracted templates are not based upon the number of inputs, they are 
dynamically reconfigured to match the number of inputs each time they are used; 



wo 0^63529 PCTAJSOl/05712 

-11- 

• Command-line software tools can be given a graphical user interface without any 
modification, recompiling, or relinking of the tool; 

• The system can automate the use of any software tool that does not require input 
through a graphical user interface, including operating system commands and 

5 tools from other vendors; 

• Inherently manual processes may be integrated into automated processing flows 
by sending e-mail instructions to operators; 

• Automated execution eliminates file handling and tracking by operators; and 

• Status and tracking capabilities support the needs of production systems. 
10 We claim: 
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1 . A method of creating a workflow procediire, comprising the steps of: 
2 defining software tools available to the procedure; 

user specifying desired input and output products based upon the available tools; 
4 reciirsiveiy reasoning backwards from the output products to the input products to 

construct a topology of tasks and intermediate products required for the procedure; and 
6 executing the tasks to carry out the procedure. 



2, The method of claim 1, further including the step of recalling a previously 
2 stored, abstracted template to assist in determining what tasks and intermediate products 
are needed to perform the procedure. 



3. The method of claim 1, including tools featuring a mix of single and 
2 multiple inputs, such that the topology is linear, tree-like, a lattice, or any permutation 
thereof 



4. The method of claim 1, further including the step of prompting a user to 
2 intervene to assist in determining what tasks and intermediate products are needed to 
perform the procedure. 



5. The method of claim 1, further including the step of automatically 
2 replicating topology segments used to process a single input to satisfy multiple input 
tasks. 
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6. The method of claim 1, further including the step of wrapping any input 
2 that can be executed without a graphical user interface in a shell script to eliminate the 

need for recompiling, relinking, or otherwise altering the input 

7. The method of claim 1, further including the step of treating an input that . 
2 requires a graphical user interface as a manual tool to eliminate the need for recompiling, 

relinking, or otherwise altering the input. 

8. The method of claim 1, further including the step of pre-validating user 
2 specifications, such that only valid tasks and intermediate products are generated for the 

procedure. 

9. The method of claim 1, further including the step of identifying the data 
2 products as data types and data classes for step of reasoning over input data requirements. 

10. The method of claim 1, fiirther including the step of identifying certain 
2 tools as accepting items of a data class, and outputting items that belong to a data class, to 

allow for the efficient description of general-purpose tools. 

11. The method of claim 1 , further including the step of representing the user 
2 interface as a tree structure. 
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12. The method of claim 11, wherein the nodes of the tree may represent 
2 conditional nodes that select one of the underlying branches. 

13. The method of claim 11, wherein the nodes of the tree may represent 
2 multiple occurrences of that item. 

14. The method of 13, wherein the number of occurrences of an item may be 
2 tied to an arbitrary range or the number of occurrences of another item. 

15. A system for creating a workflow procedure, comprising: 

2 a data and tool definition component for defining^ available software tools along 

with their inputs and outputs; 
4 a work order creation component enabling a xiser to specify desired input and 

output products using the data and available tools, the work order creation component 
6 being operative to recursively reason backwards using the output products to the input 
products to create a topology of tasks and interm.ediate products needed to perform the 
8 workflow procedure; and 

a work order execution component operative to execute the tasks, and monitor the 
1 0 status of the procedure. 
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le. The system of claim 15, further including a database for storing abstracted 
2 templates to assist in determining the tasks and intermediate products needed to perform 
the procedure. 

17. The method of claim 15, wherein tools featuring a mix of single and 
2 multiple inputs, such that the topology is linear, tree-like, a lattice, or any permutation 

thereof 

18. The method of claim 15, further including a graphical interface for 
2 prompting a user to assist in determining what tasks and intermediate products are needed 

to perform the procedure. 

19. The method of claim 15, wherein the work order creation component is 
2 further operative to automatically replicate topology segments used to process a single 

input to satisfy multiple input tasks. 
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FIGURE 4 
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FIGURES 

Recursive Reasoning Builds Workflow Management Procedure 
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INTERNATIONAL SEARCH REPORT 



Imemaiional appUcation No. 
PCT/USOl/05712 



Continuation of Item 4 of the first sheet: WORKFLOW SYSTEM AND BUILDER OFFERS IMAGE SCRIPT TOOLS ON 
INPUT DATA TYPE 



Contittuation of Box I Reason 2: Qaisi 3 recites including coots feaiuring a mix of single and muliipU inputs, such thai the 
topology is linear, tree-like, a lattice, or any permutation thereof. 

Claim 3 is unsearchable because 1) it is unclear how to apply the language of the claim to arrive at the number or inclusioa of items 
in the permutations of the mix being claimed; 2) it is unclear as to what linear, trte-Uke. or lattice refers* The specification does not 
cure the aforementioned deficiendes. 

daiffl 5 recites the step of automatically replicating topology segments. 

Claim 5 is unseardiable because it is unclear as to what repeating topology segments refers. The specification does not cure tht 

deficiency. 

Claim 9 recites the step of identifying the data products as data types and data classes for step of reasoning over input data 
requirements. 

Claim 9 is unsearchable because it is unclear as to what for step of reasoning over input data requirements refers. There is no 
antecedent basis for this limitation. The specifkaiion does not cure the deficiency. 

aaim 13 recites wherein the nodes of the tree may represent multiple occurrences of that item. 

Claim 13 is unsearchable because I) it is unclear as to what the tree is being referred to; and 2) it is unclear as to what that item 
refers to. There is no antecedent basis for this limitation. The specification does pot cure tbt aforemeoxioned deficiencies. 

Claim 14 is unsearchable because claim 14 depends on datm 13. 

Claim 17 recites wherein tools featuring a mix of single and multiple inputs, such that the topology is linear, tree-like, a lattice, or 
any pemutation thereof 

Claim 17 is unsearchable because 1) it is unclear how to ap^dy the language of the claim to arrive at the number or inclusioa of items 
in the permutations of the mix being daimed; 2) it is unclear as to what linear, tree-like, or lattice refers; 3) it is unclear as to how 
featuring ftxnhei limits the base claim. Hie specification does not cure the aforementioned deficiencies. 

Qaim 19 recites the step of automatically replicating topology segments, 

Oaun 19 is unsearchable because it is unclear as to what replicating topology segments refers. The specification does not cure the 
deficiency. 
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